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(54) [Title of the Invention] Remote 
Controller, Personal digital 
assistance and Audio/Visual Device 
(57) [Abstract] 

[Object] lb provide a remote 
controller or a personal digital 
assistance which controls 
audio/visual devices collectively 
and to improved usabilities thereof. 
[Constitution] Communication 
means, displaying means, input 
means, storage means, voice 
output means and a central 
processing unit are provided. 
Instructions or information is 
transmitted in the form of 
languages based on commands, 
statuses and specific grammars 
from/to the audio/visual devices. 
[Effect] With no thought given to a 
position of the AV device, it is 
possible to control the AV device 
easily. 



CONSTITUTION EXAMPLE OF INTELLIGENT 
AV REMOTE CONTROLLER (FIG. l) 
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[What is Claimed is] 

[Claim l] A remote controller, comprising- 

communication means by cable or wireless such as infrared rays and 
radio waves; 

display means such as liquid crystal, a Braun tube and the like! 

input means such as a device for detecting an existence and strength 
of pressing by a pen or a finger, coordinates, or a continuous time, a mouse 
and a microphone; 

storage means for storing various kinds of information; and 

a central processing unit, 

wherein a command or information is transmitted between the 
remote controller and a single or a plurality of audio/visual devices having 
communication means similar to that of the remote controller, by use of the 
communication means. 

[Claim 2] The remote controller according to claim 1, wherein the command 
for controlling the audio/visual device, information such as an image or voice, 
and operation states of the audio/visual device and the remote controller are 
transmitted or received by use of the communication means between the 
remote controller and the single or plurality of audio/visual devices having 
the communication means similar to that of the remote controller, in a form of 
languages based on a specific grammar. 

[Claim 3] The remote controller according to claim 2, wherein the remote 
controller has a conversion device or a conversion algorithm in the inside or 
outside thereof, which is for converting bidirectionally or unidirectionally the 
command, the information or the operation state according to claim 2, which 
has been transmitted or received by use of the communication means, and a 
language form based on the specific grammar. 

[Claim 4] The remote controller according to claim 1, wherein a status 
representing operation states of the single or plurality of the audio/visual 
devices or a command operating the device is transmitted or received by use 
of the communication means between the remote controller and the single or 
plurality of the audio/visual devices having communication means similar to 
that of the remote controller. 

[Claim 5] The remote controller according to claim 1, wherein image 
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information or voice information is transmitted or received by use of the 
communication means between the remote controller and the single or 
plurality of the audio/visual devices having the communication means similar 
to that of the remote controller. 

[Claim 6] The remote controller according to claim 5, wherein the single or 
plurality of the audio/visual devices having the communication means similar 
to that of the remote controller according to claim 1 compress the image 
information or the voice information to transmit it to the remote controller 
and the remote controller decodes the compressed image information or voice 
information, which has been transmitted thereto. 

[Claim 7] The remote controller according to claim 1, wherein means such as 
a camera for receiving image information or means such as a microphone for 
receiving voice information is provided. 

[Claim 8] The remote controller according to claim 7, wherein the image 
information or the voice information inputted thereto is compressed or 
uncompressed to be transmitted to the single or plurality of the audio/visual 
devices having the communication means similar to that of the remote 
controller according to claim 1 or another remote controller similar to the 
remote controller according to claim 1. 

[Claim 9] The remote controller according to one of claims 5, 6, 7 and 8, 
wherein the compressed or uncompressed image information or voice 
information is stored in the storage means or in a storage device connected to 
the outside of the remote controller according to claim 1. 

[Claim 10] The remote controller according to claim 9, wherein the 
compressed or uncompressed image information or voice information, which 
has been stored, is read out. 

[Claim 11] The remote controller according to claim 10, wherein the 
compressed image information or voice information, which has been read out, 
is decoded. 

[Claim 12] The remote controller according to claim 1, wherein displaying is 
performed by the display means based on data transmitted from the single or 
plurality of the audio/visual devices having communication means similar to 
that of the remote controller. 

[Claim 13] The remote controller according to claim 1, wherein voice output 
means such as a speaker, a voice output terminal and the like is provided. 
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[Claim 14] The remote controller according to claim 1, wherein a processing is 
executed for the input inputted by predetermined input means. 
[Claim 15] The remote controller according to claim 1, wherein a command or 
information transmitted by use of the communication means is processed. 
[Claim 16] The remote controller according to claim 1, wherein a processing is 
executed for the voice inputted by use of the input means. 

[Claim 17] The remote controller according to claim 1, wherein when a 
command or various kinds of information is transmitted by use of the 
communication means, object device designation information, which is a 
transmission object, for the audio/visual device, or for another remote 
controller similar to that of the remote controller, is added to the command or 
the various kinds of information. 

[Claim 18] The remote controller according to claim 1, wherein object device 
designation information for designating a device, which is a transmission 
object, is extracted from a command or various kinds of information, which 
has been received by use of the communication means. 

[Claim 19] The remote controller according to claim 18, wherein it is decided 
by the object device designation information extracted whether the command 
or the various kinds of information according to claim 18, which has been 
received together with the object device designation information, is processed. 
[Claim 20] The remote controller according to claim 19, wherein when it is 
decided by the object device designation information that the command or the 
various kinds of information according to claim 19 is not processed, the 
command or the various kinds of information is transmitted. 
[Claim 21] A personal digital assistance, comprising" 

communication means by cable or wireless such as infrared rays and 
radio waves; 

display means such as liquid crystal, a Braun tube and the like; 

input means such as a device for detecting an existence and strength 
of pressing by a pen or a finger, coordinates, or a continuous time, a mouse 
and a microphone; 

storage means for storing various kinds of information; and 

a central processing unit, 
wherein a command or information is transmitted by use of the 
communication means between the remote controller and a single or plurality 
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of audio/visual devices having communication means similar to that of the 
personal digital assistance. 

[Claim 22] The personal digital assistance according to claim 21, wherein the 
command for controlling the audio/visual device, information such as an 
image or voice, and operation states of the audio/visual device and the 
personal digital assistance are transmitted or received by use of the 
communication means between the remote controller and the single or 
plurality of audio/visual devices having the communication means similar to 
that of the personal digital assistance, in a form of languages based on a 
specific grammar. 

] Claim 23] The personal digital assistance according to claim 22, wherein the 
remote controller has a conversion device or a conversion algorithm in the 
inside or outside thereof, which is for converting bidirectionally or 
unidirectionally the command, the information or the operation state 
according to claim 22, which has been transmitted or received by use of the 
communication means, and a language form based on the specific grammar. 
[Claim 24] The personal digital assistance according to claim 21, wherein a 
status representing operation states of the single or plurality of the 
audio/visual devices or a command operating the device is transmitted or 
received by use of the communication means between the remote controller 
and the single or plurality of the audio/visual devices, having communication 
means similar to that of the personal digital assistance. 

[Claim 25] The personal digital assistance according to claim 21, wherein 
image information or voice information is transmitted or received by use of 
the communication means between the remote controller and the single or 
plurality of the audio/visual devices having the communication means similar 
to that of the personal digital assistance. 

[Claim 26] The personal digital assistance according to claim 21, wherein the 
single or plurality of the audio/visual devices having the communication 
means similar to that of the personal digital assistance compress the image 
information or the voice information to transmit it to the personal digital 
assistance and the personal digital assistance decodes the compressed image 
information or voice information, which has been transmitted thereto. 
[Claim 27] The personal digital assistance according to claim 21, wherein 
means such as a camera for receiving image information or means such as a 
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microphone for receiving voice information is provided. 

[Claim 28] The personal digital assistance according to claim 21, wherein the 
image information or the voice information inputted thereto is compressed or 
uncompressed to be transmitted to the single or plurality of the audio/visual 
devices having the communication means similar to that of the personal 
digital assistance according to claim 1 or another personal digital assistance 
similar to the personal digital assistance according to claim 1. 
[Claim 29] The personal digital assistance according to one of claims 25 to 28, 
wherein the compressed or uncompressed image information or voice 
information is stored in the storage means or in a storage device connected to 
the outside of the personal digital assistance according to claim 21. 
[Claim 30] The personal digital assistance according to one of claims 30, 
wherein the compressed or uncompressed image information or voice 
information, which has been stored, is read out. 

[Claim 31] The personal digital assistance according to one of claims 30, 
wherein the compressed image information or voice information, which has 
been read out, is decoded. 

[Claim 32] The personal digital assistance according to claim 21, wherein 
displaying is performed by the display means based on data transmitted from 
the single or plurality of the audio/visual devices having communication 
means similar to that of the personal digital assistance. 

[Claim 33] The personal digital assistance according to claim 21, wherein 
voice output means such as a speaker, a voice output terminal and the like is 
provided. 

[Claim 34] The personal digital assistance according to claim 21, wherein a 
processing is executed for the input inputted by predetermined input means. 
[Claim 35] The personal digital assistance according to claim 21, wherein a 
command or information transmitted by use of the communication means is 
processed. 

[Claim 36] The personal digital assistance according to claim 21, wherein a 
processing is executed for the voice inputted by use of the input means. 
[Claim 37] The personal digital assistance according to claim 21, wherein 
when a command or various kinds of information is transmitted by use of the 
communication means, object device designation information, which is a 
transmission object, which is a transmission object, for the audio/visual device, 
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for the remote controller according to claim 1, or for another personal digital 
assistance similar to the personal digital assistance, is added to the command 
or the various kinds of information. 

[Claim 38] The personal digital assistance according to claim 21, wherein 
object device designation information for designating a device, which is a 
transmission object, is extracted from a command or various kinds of 
information, which has been received by use of the communication means. 
[Claim 39] The personal digital assistance according to claim 38, wherein it is 
decided by the object device designation information extracted whether the 
command or the various kinds of information according to claim 38, which 
has been received together with the object device designation information, is 
processed. 

[Claim 40] The personal digital assistance according to claim 39, wherein 
when it is decided by the object device designation information that the 
command or the various kinds of information according to claim 39 is not 
processed, the command or the various kinds of information is transmitted. 
[Claim 41] An audio/visual device, comprising: 

communication means by cable or wireless such as infrared rays and 
radio waves, 

wherein communication with a device such as an audio/visual device 
having communication means similar to that of the audio/visually device, a 
remote controller and a personal digital assistance is performed 
bidirectionally by use of the communication means. 
[Claim 42] An audio/visual device, comprising: 

communication means by cable or wireless such as infrared rays and 
radio waves, 

wherein communication with an audio/visual device having 
communication means similar to that of the audio/visually device or a 
personal digital assistance is performed unidirectionally by use of the 
communication means. 

[Claim 43] The audio/visual device according to one of claims 41 and 42, a 
conversion device or a conversion algorithm is provided in the inside thereof 
or in the outside thereof, which is for processing a command for controlling 
the audio/visual device in a form of language based on a specific grammar, 
information such as an image and a voice, or an operation state of the 
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audio/visual device, or a remote controller or a personal digital assistance 
having communication means similar to that of the audio/visual device, which 
has been transmitted or received by use of the communication means. 
[Claim 44] The audio/visual device according to one of claims 41 and 42, 
wherein information part designating an object device is extracted from a 
command or various kinds of information designating the object device, which 
has been transmitted thereto. 

[Claim 45] The audio/visual device according to claim 44, wherein by referring 
to the information part designating the object device, it is decided whether 
the command or the various kinds of information designating the object 
device, which has been transmitted thereto, is processed or not. 
[Claim 46] The audio/visual device according to claim 45, wherein by referring 
to the information part designating the object device, when it is decided that 
the command or the various kinds of information designating the object 
device according to claim 45, which has been transmitted thereto, is not 
processed, the command or the various kinds of information designating the 
object device is transmitted to another audio/visual device, a remote controller 
or a personal digital assistance having communication means similar to that 
of the audio/visual device according to one of claims 41 and 42. 
[Detailed Description of the Invention] 
[0001] 

[Field of the Industrial Application] The present invention relates to a remote 
controller, a personal digital assistance and an audio/visual device, and 
particularly to a system (a system obtained by suitably combining the remote 
controller, the personal digital assistance and the audio/visual device) which 
is used by users to control collectively and easily audio/visual devices in their 
household or the like. 
[0002] 

[Prior Arts] Nowadays, there are various audio/visual devices including a 
television set, a VTR and the like (hereinafter referred to as AV device) in 
household or the like. Many recent AV devices are operated by a remote 
controller even from a remote place, and the number of the remote controllers 
increases inevitably with an increase in the number of the AV devices. Thus, 
there is a disadvantage that operations of the AV devise become cumbersome. 
In order to solve such problem, learning type remote controllers in which 
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functions of plural remote controllers are installed therein have already been 
put on the market. 

[0003] As a construction example of a system utilizing an infrared 
communication used in an ordinary remote controller, Japanese Patent Laid* 
Open No. Hei 2 (1990)-257731 gazette discloses means for allowing this 
system to be used also for a communication, a maintenance and a diagnosis 
by exchanging information, by constructing a radio communication network 
among a computer, a factory process controller and a handy terminal. 
[0004] 

[Subjects to be Solved by the Invention] However, users have to perform 
complicated setting operations previously for the above described learning 
type remote controller in many cases, and one button has a plurality of 
functions corresponding to AV devices which are operation objects. 
Accordingly, the above described learning type remote controller is not always 
easily operated. That is, there are many things which users must learn 
about the learning type remote controller. Furthermore, the learning type 
remote controller has a disadvantage that it is incapable of coping with too 
many AV devices in some cases, and that only a part of functions can be 
operated among operable functions of the AV devices. 

[0005] The technology disclosed in Japanese Patent Laid-Open No, Hei 
2(1990)-257731 gazette requires an infrared communication unit and a 
network device, and circumstances in which a network can be constructed by 
use of this system is limited. Specifically, to introduce this system into 
households only for watching and listening to the audience of the AV device is 
practically impossible. 

[0006] The present invention was invented in consideration of the foregoing 
respects, an object of the present invention is to provide an intelligent AV 
remote controller which is capable of collectively controlling AV devices in a 
household by communicating the AV devices bidirectionally without 
introducing an enormous amount of investment and many dedicated devices, 
and which is capable of drastically improving usability of the AV devices for 
users by having an excellent user interface. 
[0007] 

[Means for Solving the Subjects] In order to achieve the foregoing object, the 
intelligent AV remote controller according to the present invention comprises 
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a communication unit by cable or wireless including infrared rays and radio 
waves, an input unit such as a pen, a mouse, or a touch panel for detecting 
pressing by a finger, a display unit such as liquid crystal, a Braun tube, a 
voice output unit, a storage unit for storing various settings, and a control 
unit for processing and controlling various data. 
[0008] 

[Operation] In the display unit, the intelligent AV remote controller displays 
various kinds of information to a user. On the other hand, the user selects a 
predetermined area with the input unit, whereby a user's intention is 
transmitted to this intelligent AV remote controller. Furthermore, the 
communication unit exchanges various kinds of information bidirectionally 
with other AV devices. The voice output unit outputs voice to the user. By 
providing these units in the intelligent AV remote controller, it is possible to 
exchange various kinds of information and intention with the outside thereof. 
A portion storing the various kinds of information is the storage unit, and a 
portion controlling all portions described above is the control unit. 
[0009] 

[Embodiments] Embodiments of the present invention will be described below 
with reference to the accompanying drawings. Fig. 1 is a block diagram 
showing a constitution of an intelligent AV remote controller according to one 
embodiment of the present invention. 

[0010] As shown in Fig. 1, in the intelligent AV remote controller of this 
embodiment, as means for presenting information to a user, there are a 
display unit 102 such as a liquid crystal and a Braun tube, and a voice output 
unit 105 including a voice output terminal for an earphone, a head phone, a 
speaker of the like, or a built-in speaker or the like. An input unit 104 is 
provided to accept an operation of the user, and a touch panel, a mouse, a pen 
or the like is used as the input unit 104. For the convenience of the user for 
whom input by hand is difficult, input means such as a microphone and the 
like for inputting a voice may be provided in some cases. 

[00 11] In order to perform bidirectional communication with other AV devices, 
there is a communication unit 103. The communication unit 103 may adopt 
a wireless type by use of infrared rays or radio waves such as FM waves, and 
a cable type directly connected to the AV device by cables. 

[0012] There is a storage unit 106 for storing various kinds of information 
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including various settings set by a user or setting of the AV device, and a 
memory, a hard disc or the like can be used. The storage unit can be 
provided also outside the intelligent AV remote controller. 

[0013] It is a control unit 101 that controls each unit described above, 
analyzes and generates a script and data. Herein, the script describes 
procedures for controlling each element which is an operation object of the AV 
device, and is synonymous with a program. It is possible to give a feature to 
the intelligent AV remote controller, in which, by incorporating an interpreter 
for converting an interpretation and form of the script in the intelligent AV 
remote controller or providing it in the outside thereof, a communication with 
an AV device dealing with a script having a different form is enabled. 
[0014] It is possible to give general-purpose devices generally called a 
personal digital assistance the same roll as that of the above described 
intelligent AV remote controller, because the general-purpose device can 
naturally exhibit the same function as that of the intelligent AV controller as 
long as the general-purpose device has the same constitution as that of the 
intelligent AV remote controller. Herein, the personal digital assistance 
means the one that has the same constitution as that of the intelligent AV 
remote controller according to the present invention, and that can perform an 
operation other than a control by the AV device described here. Thus, an 
advantage that the AV device can be controlled by the above described 
personal digital assistance utilized for various applications is brought about. 
At this time, the personal digital assistance may be originally provided with 
such function, or the above described function is not changed also by adopting 
such a style that a program and an interface to control the AV device are 
added later on. Therefore, in the following all embodiments and drawings, 
the contents of the descriptions in this specification do not change even when 
the intelligent AV remote controller is replaced with the above described 
personal digital assistance. 

[0015] Fig. 2 is a diagram showing an example of an external appearance of 
the above described intelligent AV remote controller. In this example, as 
shown in Fig. 2, there is a liquid crystal display-cum-touch panel 201, which 
serves both as the display portion and an input portion operated by fingers or 
the like. Furthermore, in order to perform a communication with the AV 
device, an infrared interface 202, a radio wave interface 203, and a cable 
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connector 204 are provided, and the intelligent AV remote controller is 
compatible with a communication by wireless (radio wave, infrared ray) or 
cable. As described above, by providing the plurality of communication 
interfaces, an advantage is brought about, in which the intelligent AV remote 
controller can cope even when a plurality of communication means used by 
the AV device simultaneously exist. As a matter of course, the function never 
changes when only one communication interface is provided. Then, as voice 
output means, an earphone jack 205 for connecting an earphone, a head 
phone or the like, and a speaker 206 are provided. 

[0016] Fig. 3 explains the communication between the intelligent AV remote 
controller 301 and the AV device 302. The communication between the 
intelligent AV remote controller 301 and the AV device 302 means exchanges 
of a command or a status, a script and data ordinarily. The intelligent AV 
remote controller 301 and the AV device 302 perform the communication- 

(A) when the intelligent AV remote controller performs a confirmation for the 
existence and operation of the AV device (303A); 

(B) when the intelligent AV remote controller performs the control of the AV 
device (303B); 

(C) when the AV device wishes to transmit certain information to the 
intelligent AV remote controller (303C); and the like. 

[0017] First, the case of (A) corresponds to 303A of Fig. 3, and this is 
performed when an operation of the intelligent AV remote controller starts 
and performed at regular time intervals. The time at which the operation of 
the intelligent AV remote controller starts means a time at which the user 
turns on an operation start switch of the intelligent AV remote controller, or a 
time at which a predetermined operation determined so as to start the 
operation is performed. The confirmation of the existence and operation of 
the AV device, which is performed at regular time intervals, is to confirm 
whether the AV device exists within a communication range of the intelligent 
AV remote controller. Furthermore, if the AV device exists within that range, 
the confirmation thereof is to confirm what operation the AV device is 
performing. The intelligent AV remote controller is excellent in portability, 
and when the user moves while carrying the intelligent AV remote controller, 
this confirmation is means for coping with a possibility of occurrence of AV 
devices incapable of communication. However, this confirmation needs not to 
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be performed at regular time intervals, but may be performed in response to a 
certain operation. 

[0018] Next, a communication content of the foregoing 303A will be described. 
First, the intelligent AV remote controller scans around itself by use of 
communication means. Upon scanning, a response comes back from the AV 
device. In this case, procedures in which the intelligent AV remote controller 
sends out AV device confirmation command or a script, and the AV device on a 
controlled side sends out data and the like in response to this may be adopted. 
Alternatively, procedures in which the device on the controlled side sends out 
data and the like continuously or intermittently, and the intelligent AV 
remote controller receives the data and the like may be adopted. At this time, 
the data sent out from the AV device on the controlled side is a device ID of 
the AV device, a script in which procedures for controlling elements of the AV 
device that can be controlled are described, or other data. As an example of 
the device ID of the AV device, a type of the AV device, a product name, a 
maker, a manufacture number and the like are enumerated. Furthermore, 
when the AV device is a VTR, the things that can be controlled are functions 
such as a play, a fast-rewind, a fast-forward, a pause, a stop, and recording, 
and a script corresponding to these is sent from the AV device. At the same 
time, icon data for displaying buttons of the respective functions on a screen 
of the intelligent AV remote controller may be sent out from the AV device. 
As one of other data, icon data for indicating the AV device on the intelligent 
AV remote controller may be sent out. 

[0019] Next, the content of 303B of Fig. 3 in the case of the foregoing (B) will 
be described. When the user designates the operation of the AV device on the 
intelligent AV remote controller, or when the user recognizes that the 
intelligent AV remote controller is necessary, communication is performed 
between the intelligent AV remote controller and the AV device. At this time, 
an ID of the AV device that is a control object, an ID of an object that is a 
control object, an execution script to the object and data are sent out to the AV 
device from the intelligent AV remote controller. Thus, the user can operate 
the AV device. On the other hand, in response to this, a status or a script, 
which expresses a present operation state, is sent out from the AV device to 
the intelligent AV remote controller. 

[0020] Furthermore, when an image of a TV is displayed on a screen of the 
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intelligent AV remote controller, or when voice is outputted from a voice 
output device such as a speaker built in the intelligent AV remote controller 
and a head phone connected to a voice output terminal, an image and voice 
data are uncompressed or compressed, and then can be also sent out from the 
AV device to the intelligent AV remote controller solely or together with the 
foregoing operation state status. 

[0021] Finally, the content of 303C of Fig. 3 in the case of the foregoing (c) will 
be described. In the 303C, when abnormality of some kind occurs in the AV 
device, a status or a script notifying the occurrence of the abnormality is sent 
out from the AV device to the intelligent AV remote controller. When the user 
needs to know the occurrence of the abnormality, the intelligent AV remote 
controller displays the abnormality on its screen in response to the script and 
the like, whereby the user can know the occurrence of the abnormality even 
when the user is not present near the AV device causing the abnormality. 
Furthermore, when the foregoing abnormality is displayed on the screen, a 
countermeasure is displayed simultaneously on the screen, whereby a feature 
that the user can remove the abnormality without confusion can be obtained. 
The above described countermeasure may be stored in the intelligent AV 
remote controller by a way of some kind, or the countermeasure may be 
included in a script which, when the AV device notifies the abnormality, is 
sent to the intelligent AV remote controller. Furthermore, when the 
intelligent AV remote controller counters the abnormality of the AV device, 
the intelligent AV remote controller sends out a script for the countermeasure 
to the object AV device. For example, in the case where a CD is inserted into 
a CD player, when the inserted CD is not recognized because of occurrence of 
an error of some kind, the CD player makes a script in which the occurrence 
of the abnormality and the countermeasure are described, and transmits it to 
the intelligent AV remote controller. When the content of the 
countermeasure is that a tray of the CD player is once opened and closed 
again, and the recognition of the CD is tried again, the intelligent AV remote 
controller which has received the script transmits a script to the CD player, 
which notifies that the operation should be performed. At this time, the 
intelligent AV remote controller displays a message to the effect that the 
abnormality has occurred on its screen, and can encourage the user to wait. 
[0022] There are several methods to construct a system in which the 
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intelligent AV remote controller and the AV device on the controlled side are 
incorporated. First, the AV device includes the one configured to be capable 
of communicating bidirectionally between the AV devices and between the 
intelligent AV remote controller and the AV device^ and the one configured to 
be incapable of communicating between the AV devices or configured to 
perform no communication between the AV devices. Herein, the AV device 
incapable of communicating between them represents conventional AV 
devices which can be operated by a remote controller. With respect to the AV 
devices which can perform the bidirectional communication, also a system 
construction in which a part of the AV devices alone perform the bidirectional 
communication with the intelligent AV remote controller can be adopted. 
Such system construction is extremely effective when all of the AV devices are 
not in the state where they can communicate with the intelligent AV remote 
controller bidirectionally. Accordingly, 

(a) ' a case where all of the AV devices perform the bidirenctional 
communication with the intelligent AV remote controller, 

(b) • a case where a part of the AV devices perform the communication with 
the intelligent AV remote controller, 

(c) • a case where the intelligent AV remote controller constructs a system 
together with the AV devices which are incapable of performing the 
bidirectional communication are shown below. 

Note that the communication described here shall be any of ones by wireless 
such as infrared ray and radio waves or by cables. 

[0023] Fig. 4 is an example of a system construction in the case of the first 
item (a) described above, in which the AV device capable of the bidirectional 
communication and the intelligent AV remote controller construct a system. 
Such propagation paths of the information are realized when all of the AV 
devices and the intelligent AV remote controller exist in the same room. 
[0024] As the AV devices on the controlled side, a television 408 including a 
satellite broadcasting and a cable television, a stereo 409 including a CD 
player, a tuner, a cassette deck, a DAT deck, an MD deck, a DDC deck, a 
record player and an amplifier, a VTR 407, and an LD player 410 are 
enumerated. As a matter of course, existence of other AV devices 411 is not 
cared at all. Existences of the plurality of the AV devices of such kind are not 
also cared at all. In the example of Fig. 4, the stereo 409, the TV 408, the 
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VTR 407, the LD player 410 and other AV devices 411 communicate 
separately with the intelligent AV remote controller 401, and the respective 
AV devices are capable of the bidirectional communication with other AV 
devices. 

[0025] A mouse 402 and a pen 403 on the intelligent AV remote controller 401 
side are similar to a mouse and a pen in the input unit 104 of Fig. 1, and are 
provided when input means such as a touch panel is not incorporated in the 
intelligent AV remote controller. The head phone 404, the earphone 405 and 
a speaker 406, which are voice external output means, are respectively 
similar to the head phone, the earphone and the speaker in the voice output 
unit 105 of Fig. 1, and can be connected to a voice output terminal when the 
voice output terminal is provided in the intelligent AV remote controller. 
[0026] In the case of the system constitution as described in Fig. 4, since the 
intelligent AV remote controller 401 performs the communications with the 
respective AV devices, this system constitution has a feature that a time 
difference between the transmission of the information and the start of the 
operation of the AV device to which the information is transmitted among the 
respective AV devices is small. Furthermore, in this constitution, instead of 
conventional cables and the like for sending out voice and image signals, it is 
possible to transmit and receive the voice and the image signal by the 
communication among the AV devices of this system, and troublesome wirings 
among the devices are never necessary at all. This system is very easily 
usable for the user. 

[0027] Fig. 5 is an example of a system constitution in the case of the second 
item (b) described as above. Also in the case of the example of the system 
constitution shown in Fig. 5, the AV device on the controlled side may be 
diverse as in the case of Fig. 4, and may be plural. The mouse 502, the pen 
503, the head phone 504, the earphone 505 and the speaker 506 connected to 
the intelligent AV remote controller 501 are similar to the mouse, the pen, the 
head phone, the earphone and the speaker of Fig. 1, described above. Herein, 
the example is shown, in which all of the AV devices performs the 
bidirectional communications with the intelligent AV remote controller 501 
via the television 507, and the stereo 508, the LD player 509, the VTR 510, 
and other AV devices 511 perform the communications with the television 507, 
respectively. The communication means at this time may be wireless or cable 
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as described above. 

[0028] By adopting the system constitution described in Fig. 5, the following 
effects can be brought about. Even when a certain AV device is in a state 
where it cannot directly communicate with the intelligent AV remote 
controller 501, for example, even when the television 507 and the intelligent 
AV remote controller 510 exists in the same room and the stereo 508 exists in 
a different room, the intelligent AV remote controller 501 can control the 
stereo 508 via the television 507. This can be realized by the procedures in 
which a script and the like are first transmitted from the stereo 508 to the 
television 507, and the television 507 which has received the script of the 
stereo 508 transmits it to the intelligent AV remote controller 501. 
Furthermore, reverse procedures are also possible. 

[0029] Furthermore, for example, a method can be adopted, in which only the 
television 507 among the AV devices on the controlled side has an interpreter 
for interpreting the script, the script is exchanged between the intelligent AV 
remote controller 501 and the television 507, and a command used in a 
conventional remote controller is exchanged between the television 507 and 
each of other AV devices. According to this method, a feature is brought 
about, in which not all of the AV devices need to have the interpreter, and 
simplification of a circuit of the AV device can be achieved. Furthermore, as 
in the case of Fig. 4, it is also possible to transmit and receive a voice and an 
image signal by the communication of this system. 

[0030] Fig. 6 is an example of a system constitution in the case of the above 
described item (3). In Fig. 3, a system is constructed by the intelligent AV 
remote controller 601 and the AV devices of two kinds. One is incapable of 
the bidirectional communication, and the other does not perform the 
bidirectional communication. The AV device on the controlled side may be 
diverse as in the case of Fig. 4, and may be plural. In the example of Fig. 4, 
however, the stereo 607, the TV. 608, the VTR 609, the LD player 610 and 
other AV devices 611 individually perform communications with the 
intelligent AV remote controller 601. Furthermore, the mouse 602 and the 
pen 603, which are external input means of the intelligent AV remote 
controller 601, are similar to the mouse and the pen of Fig. 1, and the head 
phone 604, the earphone 605, and the speaker 606, which are voice external 
output means, are similar to the head phone, the earphone and the speaker of 
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Fig. 1 described above, respectively. 

[0031] A merit obtained by constructing the system as shown in Fig. 6 is that 
the system can be constituted together with the AV devices incapable of the 
bidirectional communication among the devices. When the system is 
constituted by the conventional AV devices operated by remote controllers and 
the intelligent AV remote controller 601, the intelligent AV remote controller 
601 operates as a comprehensive remote controller which controls the AV 
device based on data previously inputted, or as a learning type remote 
controller which comes to be capable of controlling a certain AV device by 
storing the operation of the remote controller afterward, which is present 
initially in the AV device. However, a feature different from that of a 
conventional remote controller is that the intelligent AV remote controller 601 
comprises an excellent user interface as described later. Specifically, the 
example in which the system is constituted together with the AV device 
capable of the bidirectional communication among the devices was shown in 
Fig. 4 and Fig. 5, the AV devices can be controlled with the quite same screen 
operation in this example. 

[0032] The AV device equipped with the communication function described 
above may be so called an AV device capable of an unsophisticated 
bidirectional communication, which can bidirectionally exchanges data with 
an intelligent AV remote controller, and cannot perform a communication 
with other AV devices. When a system is constituted by this AV device 
capable of the unsophisticated bidirectional communication and the 
intelligent AV remote controller, a system constitution appears to be quite the 
same as that of Fig. 6. The system constitution is configured so that the 
communication between the AV device and the intelligent AV remote 
controller includes all of the contents shown in Fig. 3, and only the exchange 
of data among the AV devices is impossible. 

[0033] It is possible to constitute a system, in which the AV device capable of 
the bidirectional communication, the AV capable of the unsophisticated 
bidirectional communication, and the AV device which is incapable of the 
bidirectional communication or do not perform the bidirectional 
communication are mixed and used as AV devices on a controlled side. Thus, 
even when the user buys a new AV device capable of the bidirectional 
communication, the user needs not to replace the conventional AV device with 
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this new one, and can continue to use the conventional AV device in this 
system. At this time, a display unit of the intelligent AV remote controller 
may show what communication the AV device can perform, or alternatively 
the display unit thereof needs not show it. In the case where the display unit 
thereof shows it, the user can know that a range where the intelligent AV 
remote controller can control the AV device is limited. In the case where the 
display unit thereof does not show it, the user can manipulate the intelligent 
AV remote controller to some extent even when the user does not have 
knowledge of the bidirectional communication, and even when user does not 
know what communication function is in the possession of an AV device which 
constitutes the system. 

[0034] Furthermore, in Fig. 4 to Fig. 6, a conventional cable for propagating a 
voice and an image information may be connected between the AV devices or 
between the AV device and the intelligent AV remote controller. This is 
effective means when it is difficult to transmit and receive the voice and the 
video data with the communication function of this system for the reason of a 
burden on a time, a quality and hardware. 

[0035] Then, in Fig. 4 to Fig. 6, by providing an interpreter, which converts 
the form of the script, in each of the AV devices or the outside thereof, it is 
possible to allow the system to possess a feature that the communication 
between the AV devices dealing with scripts having different forms and 
between the AV device and the intelligent AV remote controller is enabled. 
Furthermore, when the foregoing interpreter is incorporated in one certain 
AV device or the interpreter is solely provided, and when the system is 
configured to perform the communication between the AV devices or between 
the AV device and the intelligent AV remote controller via the AV device or the 
interpreter, other AV devices and the intelligent AV remote controller can 
perform the communication mutually with each other without providing the 
interpreter therein. 

[0036] Next, an example of the user interface of the intelligent AV remote 
controller will be shown. The feature of the user interface of the intelligent 
AV remote controller is as follows. Specifically, it is possible to set a plurality 
of methods to perform a certain manipulation of the AV device. The user 
interface can be easily understood through intuition by using metaphor. 
User's convenience can be achieved by displaying various kinds of 
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information simultaneously. A screen display can be changed in accordance 
with requirement of the user. 

[0037] Fig. 7 shows an example of an initial screen of the intelligent AV 
remote controller. Fig. 7 is a display example when responses from the 
television 701, the stereo 702, the video (l) (video deck) 703, the video (2) 
(video movie) 704, and the LD player 705 are respectively made as a result of 
a transmission of an AV device confirmation command from the intelligent AV 
remote controller through the communication means. Note that the 
respective AV devise illustrated here shall be capable of communicating with 
other AV devices and the intelligent AV remote controller bidirecitionally. 
[0038] On this screen, icons other than those of the time 706 and the help 
button 707 show ones which have been sent as data from the respective AV 
devices to the intelligent AV remote controller. Note that the icons may 
assign data originally possessed by the intelligent AV remote controller based 
on a type of the AV device. Furthermore, though the name of the AV device 
under the icon is one obtained by assigning the device name originally stored 
in the intelligent AV remote controller based on data sent from the respective 
AV devices, the name of the AV device may be data such as a product name, 
which has been sent from each AV device. Various data concerning the 
respective AV devices, which has been once transmitted, is stored in a storage 
device such as a memory and a hard disc by the intelligent AV remote 
controller. This memory may be permanent, or may be deleted at the time of 
the operation completion of the intelligent AV remote controller caused by a 
predetermined operation. 

[0039] Furthermore, the help button 707 is always displayed on the screen, 
and the manipulation and setting of the intelligent AV remote controller and 
the explanation of the manipulation method other than manipulation of the 
AV device are used if necessary. However, the details will be described later. 
The screen may always display the time 706, and the user may set that the 
time 706 is always displayed, lb display the time 706 is useful at the time a 
timer recording of the video is set. 

[0040] Fig. 8 is a block diagram of an example of an initial screen of Fig. 7 
above described. Though the intelligent AV remote controller 801 is the same 
as that which has been described until now, the pen 802 as external input 
means and the speaker 803 as voice output means are respectively provided 
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in the intelligent AV remote controller 801. As an AV device on the controlled 
side, the stereo 805, the LD player 806, the VTR (l) (video deck) 807 and the 
VTR (2) (video camera) 808 are in a state where they are capable of 
communicating with the intelligent AV remote controller 801 via the 
television 804. Specifically, they are in the same state as that shown in Fig. 5. 
[0041] Next, the case where the intelligent AV remote controller comes to be 
incapable of receiving a response from the AV device which has been 
confirmed to exist will be described. In order to use the intelligent AV remote 
controller for the same purpose as that of an ordinary remote controller, the 
user often moves while carrying it. As a result, there is a possibility that the 
intelligent AV remote controller and the AV device comes to be incapable of 
communicating with each other. This case is as follows, for example. 
Specifically, the intelligent AV remote controller and the AV device perform 
communications by radio, and radio waves, infrared ray and the like as the 
means of the wireless communication come to be incapable of reaching each of 
them. Therefore, the intelligent AV remote controller confirms that the 
intelligent AV remote controller is in a state where it is capable of 
communicating with the AV device and how each of the AV devices operates at 
present, at regular time intervals or when a predetermined operation is 
performed. 

[0042] Fig. 9 is an example of a block diagram in the case where no response 
from the AV device which has been confirmed to exist is made, as described 
above. In the example shown in Fig. 9, the VTR (2) 808 comes to be 
incapable of communicating with the intelligent AV remote controller 801 for 
the above described reason or the like in the foregoing example of the device 
constitution shown in Fig. 8, and the VTR (2) 908 is in a state where it cannot 
communicate with any of the devices. The intelligent AV remote controller 
901, the pen 902, the speaker 903, the television 904, the stereo 905, the LD 
player 906 and the VTR (l) 907 are the same as the intelligent AV remote 
controller 801, the pen 802, the speaker 803, the television 804, the stereo 805, 
the LD player 806 and the VTR (l) 807 of Fig. 8, respectively, and in the same 
states as those of them of Fig. 8. 

[0043] Fig. 10 shows how the foregoing states of Fig. 9 are reflected on the 
display of the screen of the intelligent AV remote controller. In the display 
example of the screen of Fig. 10, the situation in which no response from the 
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video (2) (video movie) 1004 which is the same as the VTR (2) 908 of Fig. 9 is 
made and a communication comes to be incapable for some reason or other is 
notified to the user visually by marking the symbol x on the icon of the video 
(2). By allowing the intelligent AV remote controller to possess such function, 
it is possible to resolve fret about no response of the AV device when the 
button is depressed, which is apt to occur in using the conventional remote 
controller. At this time, by displaying the countermeasure on the screen of 
the intelligent AV remote controller, it is possible to reduce annoyance of the 
user. Incidentally, the television 1001, the stereo 1002, the video (l) 1003, 
the LD player 1005, the time 1006 and the help button 1007 of Fig. 10 are the 
same as the television 701, the stereo 702, the video (l) 703, the LD player 
705, the time 706 and the help button 707 of Fig. 7, respectively. 
[0044] Fig. 11 shows a display example of the screen in the case where the AV 
device is operated actually on the screen of the intelligent AV remote 
controller, and here shows an example of manipulation when playback of the 
video (1) is performed. The icon of the video (l) 703 of Fig. 7, which is an 
initial screen, is selected by a predetermined pointing device such as a pen, a 
mouse and a finger. In a state where the icon of the video (l) 703 is selected, 
this selected icon is overlapped on an icon of the television 701 in which a 
playback screen is wished to be displayed. Upon overlapping the icon of the 
selected icon, the icon of the television becomes a selected state, and a series 
of operations performed by the user is shown to be effective. 
[0045] The icon of the 1101 in Fig. 11 indicates that a further selected state is 
brought about (shown by hatching in the drawing) with the icon of the video 
(l) being overlapped on the television as described above. In the portion 
where the icon of the video (l) has been originally displayed, a thing like a 
shadow 1103 of the video (l) is displayed as shown in Fig. 11, and this can 
clearly demonstrate that the video (l) is moved from the original position. As 
a matter of course, the shadow 1103 of the video (l) needs not to be displayed. 
Furthermore, an operation in which this video deck is incorporated in the 
television is metaphor that the video is displayed on the television, and a 
merit that the remote controller of the embodiment is very intuitively and 
easily understood compared to the conventional remote controller is brought 
about. By this operation of the user, a script describing the operation is sent 
from the intelligent AV remote controller to the AV device corresponding to 
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the video (l). Note that the stereo 1102, the video (2) 1104, the LD player 
1105, the time 1106 and the help button 1107 of Fig. 11 are respectively the 
same as the stereo 702, the video (2) 704, the LD player 705, the time 706 and 
the help button 707 of Fig. 7. 

[0046] Fig. 12 is an example of the display screen of the intelligent AV remote 
controller when the metaphor as described in Fig. 11 is not used. When the 
icon of the video (l) 703 of Fig. 7, which is an initial screen, is only selected by 
the predetermined pointing device such as the finger and the pen, a window 

1208 for a manipulation of the video (l) 703 is opened. The respective icons 
of the television 701, the stereo 702, the video (l) 703, the video (2) 704 and 
the LD player 705 are redrawn with sizes and at positions where they are not 
hidden by the manipulation window 1208 of the video (l). Thus, these icons 
become respectively like the television 1202, the stereo 1203, the video (l) 
1201, the video (2) 1204 and the LD player 1205, and the icon 1201 of the 
video (l) becomes in a selection state, that is, the icon is in a hatched state in 
this drawing. 

[0047] In the manipulation window 1208 of the video (l), icons of the title 

1209 of the window, the present tape counter 1210, the fast rewind button 
1211, the play button 1212, the fast forward button 1213, the pause button 
1214, the stop button 1215, the button 1216 for performing recording at once 
and the timer programmed recording 1217 are displayed. Therefore, when 
the play button 1212 is selected by the pointing device such as the pen and 
the finger, the playback of the video (l) starts. 

[0048] In the example of Fig. 12, since only the television exists as an object 
for displaying the screen, the playback screen of the video (l) is automatically 
displayed on the television. If there are the plurality of televisions, a window 
for selecting the object to display the playback screen of the video (l) may be 
opened. By adopting such system* it is possible to imitate the way how to 
operate the conventional remote controller and AV device on the intelligent 
AV remote controller, and an effect that the user is not perplexed in using the 
intelligent AV remote controller is brought about. In order to visually express 
which AV device has the window 1208, a thing like the trace 1218 of the 
opened window may be displayed. With such system, it is possible to visually 
grasp what window is opened, without reading the title 1209 of the window. 
[0049] Fig. 13 is an example of the display screen of the intelligent AV remote 
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controller when the playback of the video deck corresponding to the video (l) 
using the procedure shown in Fig. 11 and Fig. 12 is performed. In these 
drawings, the video (l) 1301, the television 1302, the stereo 1304, the video 
(2) 1305 and the LP player 1306 are the same as the video (l) 703, the 
television 701, the stereo 702, the video (2) 704 and the LD player 705 of Fig. 
7, respectively. Furthermore, this example shows a state where the 
manipulation window 1310 of the video (l) is opened so as to be capable of 
performing the manipulation of the video (l) at once. Similarly, also the 
sound level manipulation window 1309 is opened so as to be capable of 
controlling the sound level of the television. 

[0050] The user can easily know the operation state of the present video (l) by 
displaying the character 1311 expressing that the operation state is in course 
of playback and the play button with the selected state 1314 in the 
manipulation window 1310 of the video (l). At this time, by displaying the 
tape counter 1312 of the video (l), the user can know the remaining amount 
of the video tape without coming near the video deck. Furthermore, by 
displaying the present sound level numerically as well as the adjustment 
button in the sound level adjustment window 1309, the user can use this 
displaying as a guide to the sound level adjustment. 

[0051] On the other hand, among the icons of the AV devices, the video (l) 
1310 and the television 1302 are selected, that is, they are hatched. This 
shows that the video (l) and the television are in operation at present. 
Furthermore, the speaker 1303 is displayed beside the television. This shows 
that sound is comes out at present. By operating the icon 1303 of the speaker, 
it is possible to emit voice of the video (l) 1301 from other AV devices or emit 
no voice of the television 1302. Furthermore, a selection window of an AV 
device emitting voice is displayed at the time of starting a video playback, and 
the user may select this. When there are a plurality of devices displaying the 
image, a window for selecting this and an icon expressing an image display 
may be provided similarly to the icon 1303 of the speaker. 

[0052] In the window 1310 of the video 1, the fast rewind button 1313, the 
play button 1314, the fast forward button 1315, the pause button 1316, the 
stop button 1317, the button 1318 for performing the recording at once and 
the timer programmed button 1319 are the same as the fast rewind button 
1211, the play button 1212, the fast forward button 1213, the pause button 
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1214, the stop button 1215, the button 1216 for performing the recording at 
once and the timer programmed button 1217 of Fig. 12. Similarly, the time 
1307 and the help button 1308 are the same as the time 706 and the help 
button 707 of Fig. 7. 

[0053] Fig. 14 is an example of the display screen of the intelligent AV remote 
controller when the timer programmed recording setting of the video is 
performed. In Fig. 14, the video (l) 1401, the television 1402, the stereo 1403, 
the video (2) 1404 and the LD player 1405 are the same as the video (l) 703, 
the television 701, the stereo 702, the video (2) 704 and the LD player 705 of 
Fig. 7. In the example shown in Fig. 14, the window 1409 for making a 
recording schedule of the video (l) is opened, and this schedule programming 
window 1409 is opened when the user selects the schedule programming icon 
1217 in the manipulation window 1208 of the video (l) of Fig. 12. In order to 
display this on the screen, in the bottom right 1408 of the schedule 
programming window 1409, a state is displayed where the foregoing schedule 
programming icon 1217 of Fig. 12, which is hatched in this drawing, is 
selected. 

[0054] There are two large areas in the foregoing schedule programming 
window 1409. One is the area 1410 for displaying a situation where the 
schedule programming is done at present, and the other is the area 1411 for 
new schedule programming. In the former area 1410, a day, a recording start 
time, a recording finish time and a channel are described. In the latter area 
1411, a display for selecting a day, a recording staring time, a recording finish 
time and a channel is displayed. However, the remote controller of the 
present invention differs largely from the conventional remote controller in 
that it is possible to directly write a day, a time and the like, at which the 
schedule programming is intended to be performed, on the screen by use of 
the pointing device such as the pen, the mouse and the finger used in the 
intelligent AV remote controller. The written numeric characters are 
character-recognized. Furthermore, after entered line drawing is deleted, 
entering data may be displayed by printing type newly. Thus, the user can 
confirm whether the entering done by himseliTherself is effective or not. 
Furthermore, in order to make the entering simple, the mechanism 1412 for 
switching a designation of a day and a designation of a day of the week or 
morning and afternoon with one touch can be provided. Effect never change 
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even by adopting the form in which the characters are not set by writing them 
on the screen, but desired one among from the day, the day of the week, the 
time and the channel, which are already displayed in the form of a table, is 
selected. 

[0055] Furthermore, in the schedule programming window 1409 of the video 
(1) 1, there are the button 1413 for canceling the schedule programming and 
the button 1414 for finish, so as to make the cancel of the schedule 
programming and the finish easy. Note that the time 1406 and the help 
button 1407 are the same as the time 706 and the help button 707 of Fig. 7. 
[0056] Fig. 15 is one example of the display screen of the intelligent AV 
remote controller when a channel is selected for viewing the television. Fig. 
15 is the example of the screen display when the icon 701 of the television is 
selected by a predetermined pointing device in Fig. 7 showing the initial 
screen of the intelligent AV remote controller. Therefore, the icon 1502 of the 
television is in a state where it is selected, that is, in a state where it is 
hatched in this drawing. In this example shown in Fig. 15, the window 1508 
having the same role as that of the window 1309 for adjusting the sound level 
of the television of Fig. 13 and the window 1509 for selecting the channel of 
the television are opened. 

[0057] In the window 1509 for selecting the channel, the one having the same 
appearance as that of the conventional remote controller is displayed, and a 
way how to use it, it is sufficient to select a desired channel (for example, 
1510) by a predetermined pointing device. Thus, usability of it is not lower 
than that of the conventional remote controller. At this time, in the channel 
button like 1510, it is possible to provide facilities for the user by displaying 
the name of the TV station distinctively from the numerical characters 
showing the channel. The name of the TV station may be one previously 
stored in the intelligent AV remote controller, and may one set by a 
predetermined operation by the user. 

[0058] Furthermore, as one feature of the intelligent AV remote controller, a 
possibility that an image and voice data are received from the device by use of 
a communication is enumerated. In Fig. 15, by selecting the area 1511 in 
which the words "display on this screen" in the window 1509 for channel 
setting by a predetermined pointing device, the feature can be utilized. In 
other words, when the foregoing area 1511 is selected, the image and voice 
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signals are sent to the intelligent AV remote controller from the television, 
and the television can be viewed on the intelligent AV remote controller. At 
this time, the display unit 102 of the intelligent AV remote controller of Fig. 1 
is used for the image display, and the voice output unit 105 of the intelligent 
AV remote controller is used for the voice output. 

[0059] In the example of the system constitution, since each of the AV devices 
and the intelligent AV remote controller can communicate with each other, 
there is an effect that it is possible to view the television program on the 
intelligent AV remote controller even in a place where no television exists as 
long as the intelligent AV remote controller can communicate with any of the 
AV devices. At this time, the image and the voice are transmitted by the 
foregoing communication means in the order- from the television to the AV 
device; and from the AV device to the intelligent AV remote controller. States 
of the image and the voice data transmitted in such a way may be any of 
compressed and uncompressed ones. When they are compressed, though 
software or hardware for compressing is required for the AV device, and 
though software or hardware for decoding is required for the intelligent AV 
remote controller, there is a merit that a transmission time of the image and 
the voice data between the AV device and the intelligent AV remote controller 
is short. Furthermore, when they are uncompressed, there is a merit that 
circuit constitutions of the AV device and the intelligent AV remote controller 
are simpler compared to those when they are compressed. 

[0060] By providing a capture function in the intelligent AV remote controller, 
in which the television screen is stored by an instruction of the user and the 
television screen is continued to be displayed until a next instruction of the 
user, it is possible to provide facilities to the user even when information 
which the user wish acquire carefully is comes out. For the memory of the 
television screen at this time, a storage device provided in the intelligent AV 
remote controller or a storage device provided outside the intelligent AV 
remote controller can be used. 

[0061] Note that in Fig. 15, the video (l) 1501, the television 1502, the stereo 
1503, the video (2) 1504, the LD player 1505, the time 1506 and the help 
button 1507 are the same as the video (l) 703, the television 701, the stereo 
702, the video (2) 704, the LD player 705, the time 705 and the help button 
707 of Fig. 7, respectively. 
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[0062] Fig. 16 is an example when the television program is viewed on the 
intelligent AV remote controller. Herein, in order to make it easier to display 
the television screen, the intelligent AV remote controller is horizontally 
oriented. On the screen of the intelligent AV remote controller, the television 
screen 1604, the time 1603, the volume setting window 1601, the channel 
selection window 1602, the television view cancel button 1605 and the help 
button 1606 are displayed. 

[0063] The volume setting window 1601 is the same as the volume setting 
window 1309 of Fig. 13, and by selecting a button in the channel selection 
window 1602 with a predetermined pointing device, a channel which is 
wished to be viewed can be freely changed. At this time, in addition to the 
button to change the channel, facilities may be provided to the user by 
displaying a channel number of a TV station which is currently being 
displayed on the intelligent AV remote controller. Then, when the cancel 
button 1605 is selected in a state of Fig. 16, the screen of the intelligent AV 
remote controller returns to the state of Fig. 7 or Fig. 15. In addition to these, 
by displaying optional information such as operation situations of other AV 
devices, the control of the AV device can be easily performed. Note that in 
Fig. 16, the time 1603 and the help button 1606 are the same as the time 706 
and the help button 707 of Fig. 7, respectively. 

[0064] Fig. 17 is an example of the screen display when the help button is 
selected on the initial screen of the intelligent AV remote controller of Fig. 7. 
In the lower right; of the screen in this example, the help button 1707 is 
currently selected, that is, the help button 1707 is in a hatched state on this 
screen. Furthermore, since the help button 1707 is selected, the window 
1708 for setting is also opened. Areas for explaining a manipulation method 
of the intelligent AV remote controller, setting of the television channel and 
the TV station and other settings are displayed in the setting window 1708, 
and when the selection is performed by the predetermined pointing device as 
described above, windows for these will also be opened. To complete this 
manipulation, a finish button can be displayed in this window 1708. With 
such constitution, the user can manipulate the setting of the television 
channel which has been troublesome in the conventional system, and by 
providing a similar help button for setting the television channel, the user can 
refer to an instruction manual directly on the screen when the user does not 
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know an operational procedure. Accordingly, the intelligent AV remote 
controller is very multifunctional compared to the conventional remote 
controller, and can be simplified. 

[0065] In Fig. 17, the video (l) 1701, the television 1702, the stereo 1703, the 
video (2) 1704, the LD player 1705 and the time 1706 play the same role as 
those of the video (l) 703, the television 701, the stereo 702, the video (2) 704, 
the LD player 705 and the time 706 of Fig. 7, respectively. 

[0066] As described above, the intelligent AV remote controller has a feature 
that it can process the data displayed on the screen. Therefore, for the 
convenience of the user's use, the icons can be arranged on the screen freely, 
and the screen can be rotated optionally as shown in Fig. 16. It is possible to 
provide convenience to the user. 

[0067] When the power source of the intelligent AV remote controller is cut by 
a predetermined operation, or when the user performs a predetermined 
operation, the intelligent AV remote controller may transmit a command and 
script of power source cutting to the respective AV devices. Thus, by cutting 
the power source of the intelligent AV remote controller, power sources of all 
of the AV devices communicating with the intelligent AV remote controller can 
be cut at a time. Accordingly, the user can collectively manage the power 
sources of all of the AV devices without complicated manipulations. 
[0068] Herein, when the plurality of intelligent AV remote controller exist, it 
is expected that an instruction and a script, which have different contents 
respectively, are transmitted to the same AV device from the respective 
intelligent AV remote controllers. In order to avoid this, priorities are 
assigned to the plurality of intelligent AV remote controller by the user's 
manipulation, and when the AV device receives instruction and the like which 
cannot exist simultaneously, the AV device accepts the instruction from the 
intelligent AV remote controller having a higher priority. It is possible to 
prevent an operation content of the AV device from changing rapidly and an 
erroneous operation of the AV device. 
[0069] 

[Effects of the Invention] As described above, according to the present 
invention, the system is constituted of the intelligent AV remote controller or 
the personal digital assistance, and the AV device which has communication 
means similar to that of the intelligent AV remote controller or the personal 
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digital assistance, and which can mutually exchange the script, that is, the 
instruction and the information in the language form based on the specified 
grammar by this communication means, and thus, the user can control 
simply and collectively the AV device. By providing the interpreter, which 
interprets and generates the script, in the outside of the intelligent AV remote 
controller or the personal digital assistance, it is possible to obtain the effect 
that the instruction and the information can be mutually transmitted 
between the intelligent AV remote controller or the personal digital assistance 
and the AV device, which deal with different forms of the script. 
[0070] By performing various processings for the instruction and the 
information obtained by the intelligent AV remote controller or the personal 
digital assistance by the foregoing method, an user interface corresponding to 
various cases can be constructed, and it is possible to provide facilities for the 
user as circumstances demand. 

[0071] In the constitution of the intelligent AV remote controller or the 
personal digital assistance according to the present invention, when the 
image signal in a compressed or uncompressed state is transmitted from the 
AV device to the intelligent AV remote controller or the personal digital 
assistance in addition to the foregoing script, there is an effect that the image 
signal can be seen on the screen of the intelligent AV remote controller or the 
personal digital assistance. At this time, by providing the internal speaker or 
the voice output terminal in the intelligent AV remote terminal or the 
personal digital assistance, it is possible to obtain a feature that the user can 
listen to the voice in the compressed or uncompressed state, which has been 
transmitted thereto together with the image signal. Furthermore, the image 
or the voice signal in the compressed state, which has been transmitted 
thereto, is temporarily stored in the storage device incorporated in the 
intelligent AV remote controller or the personal digital assistance or 
connected to the outside thereof, whereby an effect appears in which it is 
possible for the intelligent AV remote controller or the personal digital 
assistance having no function of decoding it in real time to take a 
countermeasure. 

[0072] The intelligent AV remote controller or the personal digital assistance 
of the present invention can control the AV device which can be controlled by 
the conventional remote controller. Accordingly, an effect is exhibited, in 
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which the user needs not to buy a new AV device when the user uses the 
intelligent AV remote controller or the personal digital assistance. 
[0073] By providing a function in the intelligent AV remote controller or the 
personal digital assistance, in which an instantaneous image and voice are 
stored and they are continuously reproduced, even when the user is suddenly 
motivated by necessity to retain information of the television or the like, the 
user can satisfy the need quickly and simply 
[Brief Description of the Drawings] 

[Figure l] Fig. 1 is a block diagram showing a constitution of an intelligent AV 
remote controller according to an embodiment of the present invention. 
[Figure 2] Fig. 2 is a perspective view showing an example of an appearance 
of the intelligent AV remote controller according to the embodiment of the 
present invention. 

[Figure 3] Fig. 3 is an explanatory view showing an example of a content 
communicated between an AV device and the intelligent AV remote controller 
according to the embodiment of the present invention. 

[Figure 4] Fig. 4 is a view of a system constitution constituted by the AV 
devices and the intelligent AV remote controller according to the embodiment 
of the present invention when the AV devices communicate with the 
intelligent AV remote controller individually, and when the AV devices 
communicate with each other. 

[Figure 5] Fig. 5 is a view of a system constitution constituted by the AV 
devices and the intelligent AV remote controller according to the embodiment 
of the present invention when the AV device communicates with a television, 
or when the television communicates with the intelligent AV remote 
controller (when the AV device communicates with the intelligent AV remote 
controller via the television). 

[Figure 6] Fig. 6 is a view of a system constitution constituted by the AV 
devices and the intelligent AV remote controller according to the embodiment 
of the present invention when the AV devices communicates with the 
intelligent AV remote controller individually, and when the AV devices do not 
communicate with each other, 

[Figure 7] Fig. 7 is an explanatory view showing an example of an initial 
screen displayed in the intelligent AV remote controller according to the 
embodiment of the present invention. 



33 



Ibkkaihei 7-336778 



[Figure 8] Fig. 8 is a view of a system constitution showing one example of a 
device constitution when the screen of Fig. 7 is displayed. 

[Figure 9] Fig. 9 is a view of a system constitution showing a state where a 
communication between a video (2) and the intelligent AV remote controller 
becomes unable in Fig. 8. 

[Figure 10] Fig. 10 is an explanatory view showing one example of a display 
screen of the intelligent AV remote controller in a state shown in Fig. 9. 
[Figure 11] Fig. 11 is an explanatory view showing one example of a display 
screen of the intelligent AV remote controller when a playback of a video (l) is 
performed in Fig. 7. 

[Figure 12] Fig. 12 is an explanatory view showing another example of a 
display screen of the intelligent AV remote controller when a playback of a 
video (l) is performed in Fig. 7. 

[Figure 13] Fig. 13 is an explanatory view showing one example of a display 
screen of the intelligent AV remote controller when a playback of the video (l) 
was performed in Fig. 11 and Fig. 12. 

[Figure 14] Fig. 14 is an explanatory view showing one example of a display 
screen of the intelligent AV remote controller when a timer programmed 
recording of the video (l) was performed in Fig. 12. 

[Figure 15] Fig. 15 is an explanatory view showing one example of a display 
screen of the intelligent AV remote controller when a television channel is 
selected in Fig. 7. 

[Figure 16] Fig. 16 is an explanatory view showing one example of a display 
screen of the intelligent AV remote controller when a television channel 
wished to be viewed is selected, and when a screen of the selected channel is 
displayed on the intelligent AV remote controller in Fig. 15. 
[Figure 17] Fig. 17 is an explanatory view showing one example of a display 
screen of the intelligent AV remote controller when a help button is selected 
in Fig. 7 and other drawings. 
[Explanations of Reference Numerals] 

101 control unit of intelligent AV remote controller 

102 display unit of intelligent AV remote controller 

103 communication unit of intelligent AV remote controller 

104 input unit of intelligent AV remote controller 

105 voice output unit of intelligent AV remote controller 
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106 storage unit of intelligent AV remote controller 

201 input-cum-display unit of intelligent AV remote controller 

202, 203, 204 communication interface of intelligent AV remote controller 

205, 206 voice output unit of intelligent AV remote controller 

303A, 303B, 303C content of communication between intelligent AV remote 

controller and AV device 

402, 403, 502, 503, 602, 603, 802, 902 input means provided outside 
intelligent AV remote controller 

404, 405, 406, 504, 505, 506, 604, 605, 606, 803, 903 voice output means 
provided outside intelligent AV remote controller 

701 to 705, 1001 to 1005, 1101 to 1105, 1201 to 1205, 1301 to 1306, 1401 to 
1405, 1501 to 1505, 1701 to 1705 icon of AV device displayed on intelligent AV 
remote controller 

1208, 1310, 1409 video manipulation window 
1309, 1508, 1601 sound level adjustment window 
1509, 1602 channel manipulation window of television 
1708 help window displayed when help button is selected 
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(72) Inventor: Tatushi Yokozawa 

292 Yoshida-cho, Tbtsuka-ku, Yokohama-city, Kanagawa Prefecture 
in Kabushiki Kaisha Hitachi Seisakusyo Video Media Laboratory 
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FIG. 1 

CONSTITUTION EXAMPLE OF INTELLIGENT AV REMOTE 
CONTROLLER (FIG. 1) 

102 DISPLAY UNIT 
LIQUID CRYSTAL 
BRAUNTUBE 

101 CONTROL UNIT 

103 COMMUNICATION UNIT 
INFRARED RAY 

RADIO WAVE 
CABLE 

104 INPUT UNIT 
TOUCH PANEL 

MOUSE 
PEN 

MICROPHONE 
105 VOICE OUTPUT UNIT 
SPEAKER 

VOICE OUTPUT TERMINAL 
(EAR PHONE, HEAD PHONE AND THE LIKE) 
106 STORAGE UNIT 
MEMORY 
HARD DISC 

FIGURE 2 

EXAMPLE OF APPEARANCE OF INTELLIGENT AV REMOTE 
CONTROLLER (FIG. 2) 

201 LIQUID CRYSTAL DISPLAY-CUM TOUCH PANEL 

202 INFRARED INTERFACE 

203 RADIO WAVE INTERFACE 

204 CABLE CONNECTOR 
205 EAR PHONE JACK 

206 SPEAKER 

FIGURE 3 
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COMMUNICATION BETWEEN INTELLIGENT AV REMOTE 
CONTROLLER AND AV DEVICE (FIG. 3) 
301 INTELLIGENT AV REMOTE CONTROLLER 
302 AV DEVICE 

303A AV DEVICE CONFIRMATION COMMAND 
(OR SCRIPT) 
DEVICE ID 
ICON DATA 

SCRIPT AND DATA OF OPERABLE OBJECT 
303B OBJECT DEVICE ID 

OBJECTIVE OBJECT 
EXECUTION SCRIPT AND DATA 
IMAGE/VOICE DATA 

STATUS INDICATING PRESENT SITUATION 
(OR SCRIPT) 
303C COUNTERMEASURE SCRIPT 

STATUS INDICATING OCCURRENCE OF ABNORMALITY 

(QRSCRIPT) 
SCRIPT FOR COUNTERMEASURE 
FIGURE 4 

EXAMPLE OF SYSTEM CONSTITUTION (l) 
(EXAMPLE IN WHICH AV DEVICES COMMUNICATE 
INDIVIDUALLY WITH INTELLIGENT AV REMOTE CONTROLLER) (FIG. 



CONTROLLED SIDE 
409 STEREO 410 LD 
408 TV 411 OTHER AV DEVICES 
407 VTR 

402 MOUSE 403 PEN 401 INTELLIGENT AV REMOTE 
CONTROLLER 

EXTERNAL INPUT MEANS 

404 HEADPHONE 405 EARPHONE 

405 EARPHONE 

VOICE EXTERNAL OUTPUT MEANS 406 SPEAKER 
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FIG. 5 

EXAMPLE OF SYSTEM CONSTITUTION (2) 

(EXAMPLE IN WHICH AV DEVICES COMMUNICATE WITH 
INTELLIGENT AV REMOTE CONTROLLER VIA TV) (FIG. 5) 

508 STEREO 509 LD 510 VTR 511 OTHER AV 
DEVICES 

507 TV 
CONTROLLED SIDE 

502 MOUSE 503 PEN EXTERNAL INPUT MEANS 
501 INTELLIGENT AV REMOTE CONTROLLER 
504 HEADPHONE 505 EARPHONE VOICE EXTERNAL 
OUTPUT MEANS 

506 SPEAKER 

FIG. 6 

EXAMPLE OF SYSTEM CONSTITUTION (3) (EXAMPLE IN 
WHICH AV DEVICES COMMUNICATE INDIVIDUALLY WITH 
INTELLIGENT AV REMOTE CONTROLLER) (FIG. 6) 

CONTROLLED SIDE 

607 STEREO 608 TV 609 VTR 610 LD 611 OTHER 
AV DEVICES 

602 MOUSE 603 PEN EXTERNAL INPUT MEANS 
601 INTELLIGENT AV REMOTE CONTROLLER 
604 HEADPHONE 605 EARPHONE 606 SPEAKER 
VOCE EXTERNAL OUTPUT MEANS 

FIG. 7 

EXAMPLE OF USER INTERFACE (INITIAL SCREEN) (FIG. 7) 
701 TELEVISION 702 STEREO 703 VIDEO 1 
704 VIDEO 2 

FIG. 8 

EXAMPLE OF SYSTEM CONSTITUTION (FIG. 8) 
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805 STEREO 806 LD 807 VTR 1 808 VTR2 

804 TV 

CONTROLLED SIDE 

802 PEN 801 INTELLIGENT AV REMOTE 
CONTROLLER 

EXTERNAL INPUT MEANS 

803 SPEAKER 

VOICE EXTERNAL OUTPUT MEANS 
FIG. 9 

EXAMPLE OF DEVICE CONSTITUTION IN WHICH 
RECOGNITION BECOMES IMPOSSIBLE AT SOME MIDPOINT) (FIG. 9) 
903 STEREO 905 LD 907 VTR 1 908 VTR 2 

904 TV 
CONTROLLED SIDE 
902 PEN 

EXTERNAL INPUT MEANS 

901 INTELLIGENT AV REMOTE CONTROLLER 
903 SPEAKER 
VOICE EXTERNAL OUTPUT MEANS 

FIG. 10 

EXAMPLE OF USER INTERFACE (EXAMPLE WHEN 
RECOGNITION BECOMES IMPOSSIBLE AT SOME MIDPOINT) (FIG. 10) 

1001 TELEVISION 1002 STEREO 1003 VIDEO 1 
1004 VIDEO 2 

1005 LD 

FIG. 11 

EXAMPLE OF USER INTERFACE (EXAMPLE WHEN VIDEO 
PLAYBACK STARTS) (FIG. 11) 

1101 TELEVISION 1102 STEREO 1103 VIDEO 1 1104 
VIDEO 2 
1105 LD 
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FIG. 12 

EXAMPLE OF USER INTERFACE (ANOTHER EXAMPLE OF VIDEO 

PLAYBACK) (FIG. 12) 
1211 FAST REWIND 1212 PLAY 1213 FAST FORWARD 
1214 PAUSE 1215 STOP 1216 IMMEDIATE RECORDING 

1217 TIMER PROGRAMMED 1201 VIDEO 1 1202 TELEVISION 

1203 STEREO 1204 VIDEO 2 1205 LD 

FIG. 13 

EXAMPLE OF USER INTERFACE (EXAMPLE IN VIDEO 
REPRODUCTION) (FIG. 13) 

1309 SOUND LEVEL OF TELEVISION 
1310 VIDEO 1 
1311 PLAYING 

1313 FAST REWIND 1314 PLAY 1315 FAST FORWARD 1316 
PAUSE 1317 STOP 1318 IMMEDIATE RECORDING 1319 
TIMER PROGRAMMED 1301 VIDEO 1 1302 TELEVISION 
1304 STEREO 1305 VIDEO 2 1306 LD 

FIG. 14 

EXAMPLE OF USER INTERFACE (EXAMPLE OF VIDEO TIMER 
PROGRAMMED RECORDING) (FIG. 14) 
1409 TIMER PROGRAMMED RECORDING OF VIDEO 1 
START FINISH 
EVERY SATURDAY 
MONTH DAY 
AM HOUR MINUTE 
PM HOUR MINUTE 
1413 CANCEL 1414 FINISH 
1401 VIDEO 1 1402 TELEVISION 
1403 STEREO 1404 VIDEO 2 
1405 LD 
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FIG. 15 

EXAMPLE OF USER INTERFACE (EXAMPLE IN VIEWING 
TELEVISION) (FIG. 15) 

1508 SOUND LEVEL OF TELEVISION 
TELEVISION 

1511 DISPLAY ON TfflS SCREEN 

1501 VIDEO 1 1502 TELEVISION 1503 STEREO 1504 
VIDEO 2 1505 LD 

FIG. 16 

EXAMPLE OF USER INTERFACE (EXAMPLE WHEN TELEVISION 
SCREEN IS DIRECTLY VIEWED) (FIG. 16) 
1601 SOUND LEVEL 
CHANNEL 
1605 DISMISS 

FIG. 17 

EXAMPLE OF USER INTERFACE (EXAMPLE OF HELP/SETTING 
SCREEN) (FIG. 17) 
SETTING ETC 

EXPLANATION OF MANIPULATION METHOD 
SETTING OF TV CHANNEL 
OTHER SETTINGS 
END 

1701 VIDEO 1 

1702 TELEVISION 
1703 STEREO 

1704 VIDEO 2 
1705 LD 
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CONSTITUTION EXAMPLE OF INTELLIGENT 
AV REMOTE CONTROLLER (FIG. l) 



EXAMPLE OF APPEARANCE OF INTELLIGENT AV 
REMOTE CONTROLLER (FIG. 2) 
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EXAMPLE OF SYSTEM CONSTITUTION (2) 

(EXAMPLE IN WHICH AV DEVICES 
COMMUNICATE WITH INTELLIGENT AV 
REMOTE CONTROLLER VIA TV) (FIG. 5) 
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EXAMPLE OF SYSTEM CONSTITUTIONS) 

' TeXAMPLE IN WHICH AVDBVIC&S 
' COMMUNICATE INDIVIDUALLY WITH 
INTELLIGENT AV REMOTE CONTROLLER) 
(FIG.6) 
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COMMUNICATION BETWEEN INTELLIGENT 
AV REMOTE CONTROLLER AND AV DEVICE 
(FIG, 3) 



301 - 



INTELLIGENT AV 
REMOTE CONTROLLER 



303A 



AV DEVICE 
CONFIRMATION 
COMMAND 
(OR SCRIPT) 



OBJECT DEVICE ID 
OBJECTIVE OBJECT 
EXECUTION SCRIPT 
AND DATA 



3Q9C 



COUNTERMEASU- 
RE SCRIPT 



30S- 



DEVICE ID 
ICON DATA 
SCRIPT AND DATA OF 
OPERABLE OBJECT 



IMAGE/VOICE DATA 
STATUS INDICATING 
PRESENT SITUATION 
(OR SCRIPT) 



STATUS INDICATING 
OCCURRENCE OF ABNORMALITY 
(ORSGttPD 

SCRIPT FOR COUNTERMEASURE 



AV 
DEVICE 



Fig.4 

EXAMPLE OF SYSTEM CONSTITUTION (l) 
(EXAMPLE IN WHICH AV DEVICES 
COMMUNICATE INDIVIDUALLY WITH 
INTELLIGENT AV REMOTE CONTROLLER) 
(FIG. 4) 
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EXAMPLE OF USER INTERFACE 
(INITIAL SCREEN) (FIG. 7) 
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EXAMPLE OF SYSTEM CONSTITUTION 
(FIG.8) ' "* 
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EXAMPLE OF DEVICE CONSTITUTION IN 

WHICH RECOGNITION BECOMES 
IMPOSSIBLE AT SOME MIDPOINT) (FIG. 9) 
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EXAMPLE OF USER INTERFACE 
(EXAMPLE WHEN RECOGNITION 
BECOMES IMPOSSIBLE AT SOME 
MIDPOINT) (FIG. 10) 
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EXAMPLE OF USER INTERFACE 
(EXAMPLE WHEN VIDEO PLAYBACK 
STARTS) (FIG. 11) 
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EXAMPLE OF USfcR INTERFACE 
(ANOTHER EXAMPLE OF VIDEO 
PLAYBACK) (FIG. 12) 
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EXAMPLE OF USER INTERFACE 
(EXAMPLE IN VIDEO REPRODUCTION) 
(FIG. 13) 
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EXAMPLE OF USER INTERFACE 
(EXAMPLE OF VIDEO TIMER 
PROGRAMMED RECORDING) (FIG. 14) 
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EXAMPLE OF USER INTERFACE 
(EXAMPLE IN VIEWING TELEVISION) 
(FIG. 15) 
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EXAMPLE OF USER INTERFACE 
(EXAMPLE WHEN TELEVISION SCREEN 
IS DIRECTLY VIEWED) (FIG. 16) 
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EXAMPLE OF USER INTERFACE 
(EXAMPLE OF HELP/SETTING SCREEN) 
(FIG. 17) 



1706 



1708 - 



SETTING ETC 



(EXPLANATION OF\ 
. MANIPULATION I 



c 



SETTING OF TV 
CHANNEL 



J 



OTHER SETTINGS 



a- 



Sir' 




mi 



1702 



1703 



1704 



1705 



1707 



